Color Constancy Research Articles

Color constancy mechanisms in virtual reality environments.

Prior research has demonstrated high levels of color constancy in real-world scenarios featuring single light sources, extensive fields of view, and prolonged adaptation periods. However, exploring the specific cues humans rely on becomes challenging, if not unfeasible, with actual objects and lighting conditions. To circumvent these obstacles, we employed virtual reality technology to craft immersive, realistic settings that can be manipulated in real time. We designed forest and office scenes illuminated by five colors. Participants selected a test object most resembling a previously shown achromatic reference. To study color constancy mechanisms, we modified scenes to neutralize three contributors: local surround (placing a uniform-colored leaf under test objects), maximum flux (keeping the brightest object constant), and spatial mean (maintaining a neutral average light reflectance), employing two methods for the latter: changing object reflectances or introducing new elements. We found that color constancy was high in conditions with all cues present, aligning with past research. However, removing individual cues led to varied impacts on constancy. Local surrounds significantly reduced performance, especially under green illumination, showing strong interaction between greenish light and rose-colored contexts. In contrast, the maximum flux mechanism barely affected performance, challenging assumptions used in white balancing algorithms. The spatial mean experiment showed disparate effects: Adding objects slightly impacted performance, while changing reflectances nearly eliminated constancy, suggesting human color constancy relies more on scene interpretation than pixel-based calculations.

Learning to Remove Rain in Video With Self-Supervision.

In heavy rain video, rain streak and rain accumulation are the most common causes of degradation. They occlude background information and can significantly impair the visibility. Most existing methods rely heavily on the synthetic training data, and thus raise the domain gap problem that prevents the trained models from performing adequately in real testing cases. Unlike these methods, we introduce a self-learning method to remove both rain streaks and rain accumulation without using any ground-truth clean images in training our model, which consequently can alleviate the domain gap issue. The main idea is based on the assumptions that (1) adjacent clean frames can be aligned or warped from one frame to another frame, (2) rain streaks are distributed randomly in the temporal domain, (3) the rain streak/accumulation related variables/priors can be inferred reliably from the information within the images/sequences. Based on these assumptions, we construct an augmented Self-Learned Deraining Network (SLDNet+) to remove both rain streaks and rain accumulation by utilizing temporal correlation, consistency, and rain-related priors. For the temporal correlation, our SLDNet+ takes rain degraded adjacent frames as its input, aligns them, and learns to predict the clean version of the current frame. For the temporal consistency, a new loss is designed to build a robust mapping between the predicted clean frame and non-rain regions from the adjacent rain frames. For the rain-streak-related prior, the rain streak removal network is optimized jointly with motion estimation and rain region detection; while for the rain-accumulation-related prior, a novel non-local video rain accumulation removal method is developed to estimate the accumulation-lines from the whole input video and to offer better color constancy and temporal smoothness. Extensive experiments show the effectiveness of our approach, which provides superior results compared with the existing state of the art methods both quantitatively and qualitatively. The source code will be made publicly available at: https://github.com/flyywh/CVPR-2020-Self-Rain-Removal-Journal.

Dual-Discriminator Generative Adversarial Network with Uniform Color Information Extraction for Color Constancy

Dual-Discriminator Generative Adversarial Network with Uniform Color Information Extraction for Color Constancy

Improving RGB illuminant estimation exploiting spectral average radiance.

We introduce a method that enhances RGB color constancy accuracy by combining neural network and k-means clustering techniques. Our approach stands out from previous works because we combine multispectral and color information together to estimate illuminants. Furthermore, we investigate the combination of the illuminant estimation in the RGB color and in the spectral domains, as a strategy to provide a refined estimation in the RGB color domain. Our investigation can be divided into three main points: (1)identify the spatial resolution for sampling the input image in terms of RGB color and spectral information that brings the highest performance; (2)determine whether it is more effective to predict the illuminant in the spectral or in the RGB color domain, and finally, (3)assuming that the illuminant is in fact predicted in the spectral domain, investigate if it is better to have a loss function defined in the RGB color or spectral domain. Experimental results are carried out on NUS: a standard dataset of multispectral radiance images with an annotated spectral global illuminant. Among the several considered options, the best results are obtained with a model trained to predict the illuminant in the spectral domain using an RGB color loss function. In terms of comparison with the state of the art, this solution improves the recovery angular error metric by 66% compared to the best tested spectral method, and by 41% compared to the best tested RGB method.

Nighttime color constancy using robust gray pixels.

Color constancy is a basic step for achieving stable color perception in both biological visual systems and the image signal processing (ISP) pipeline of cameras. So far, there have been numerous computational models of color constancy that focus on scenes under normal light conditions but are less concerned with nighttime scenes. Compared with daytime scenes, nighttime scenes usually suffer from relatively higher-level noise and insufficient lighting, which usually degrade the performance of color constancy methods designed for scenes under normal light. In addition, there is a lack of nighttime color constancy datasets, limiting the development of relevant methods. In this paper, based on the gray-pixel-based color constancy methods, we propose a robust gray pixel (RGP) detection method by carefully designing the computation of illuminant-invariant measures (IIMs) from a given color-biased nighttime image. In addition, to evaluate the proposed method, a new dataset that contains 513 nighttime images and corresponding ground-truth illuminants was collected. We believe this dataset is a useful supplement to the field of color constancy. Finally, experimental results show that the proposed method achieves superior performance to statistics-based methods. In addition, the proposed method was also compared with recent deep-learning methods for nighttime color constancy, and the results show the method's advantages in cross-validation among different datasets.

Gate-controlled gain tuning of fully depleted silicon-on-insulator-based 1 T pixel for in-sensor white balance

White balance (WB) is a critical back-end processing function in image sensors to keep color constancy under various lighting conditions by adjusting the RGB color channel gain values. In this work, we proposed a novel gate-controlled gain-tuning method for a fully depleted silicon-on-insulator (FDSOI) one-transistor (1 T) pixel to achieve WB inside the sensor. Based on the pixel structure of a p-well under the buried oxide for sensing and an n-type nMOSFET on the top for readout, the 1 T pixel output gain is modulated by the MOSFET gate according to the transistor transfer characteristics. About 5x gain modulation range in RGB spectrum photoresponse (nonlinearity < 3%) is experimentally demonstrated in the devices fabricated by 22 nm FDSOI-based technology. The scheme for in-sensor WB demonstration is provided with a novel 1 T pixel array design, and the evaluation result shows in-sensor WB achieving an almost equivalent performance (Delta-E deviation < 1) compared with using conventional back-end WB.

Color constancy in real-world settings.

Color constancy denotes the ability to assign a particular and stable color percept to an object, irrespective of its surroundings and illumination. The light reaching the eye confounds illumination and spectral reflectance of the object, making the recovery of constant object color an ill-posed problem. How good the visual system is at accomplishing this task is still a matter of heated debate, despite more than a 100 years of research. Depending on the laboratory task and the specific cues available to observers, color constancy was found to be at levels ranging between 15% and 80%, which seems incompatible with the relatively stable color appearance of objects around us and the consistent usage of color names in real life. Here, we show close-to-perfect color constancy using real objects in a natural task and natural environmental conditions, chosen to mimic the role of color constancy in everyday life. Participants had to identify the color of a (non-present) item familiar to them in an office room under five different experimental illuminations. They mostly selected the same colored Munsell chip as their match to the absent object, even though the light reaching the eye in each case differed substantially. Our results demonstrate that color constancy under ideal conditions in the real world can indeed be exceptionally good. We found it to be as good as visual memory permits and not generally compromised by sensory uncertainty.

Effective cross-sensor color constancy using a dual-mapping strategy.

Deep neural networks (DNNs) have been widely used for illuminant estimation, which commonly requires great efforts to collect sensor-specific data. In this paper, we propose a dual-mapping strategy-the DMCC method. It only requires the white points captured by the training and testing sensors under a D65 condition to reconstruct the image and illuminant data, and then maps the reconstructed image into sparse features. These features, together with the reconstructed illuminants, were used to train a lightweight multi-layer perceptron (MLP) model, which can be directly used to estimate the illuminant for the testing sensor. The proposed model was found to have performance comparable to other state-of-the-art methods, based on the three available datasets. Moreover, the smaller number of parameters, faster speed, and not requiring data collection using the testing sensor make it ready for practical deployment. This paper is an extension of Yue and Wei [Color and Imaging Conference (2023)], with more detailed results, analyses, and discussions.

Synthetic Document Images with Diverse Shadows for Deep Shadow Removal Networks.

Shadow removal for document images is an essential task for digitized document applications. Recent shadow removal models have been trained on pairs of shadow images and shadow-free images. However, obtaining a large, diverse dataset for document shadow removal takes time and effort. Thus, only small real datasets are available. Graphic renderers have been used to synthesize shadows to create relatively large datasets. However, the limited number of unique documents and the limited lighting environments adversely affect the network performance. This paper presents a large-scale, diverse dataset called the Synthetic Document with Diverse Shadows (SynDocDS) dataset. The SynDocDS comprises rendered images with diverse shadows augmented by a physics-based illumination model, which can be utilized to obtain a more robust and high-performance deep shadow removal network. In this paper, we further propose a Dual Shadow Fusion Network (DSFN). Unlike natural images, document images often have constant background colors requiring a high understanding of global color features for training a deep shadow removal network. The DSFN has a high global color comprehension and understanding of shadow regions and merges shadow attentions and features efficiently. We conduct experiments on three publicly available datasets, the OSR, Kligler's, and Jung's datasets, to validate our proposed method's effectiveness. In comparison to training on existing synthetic datasets, our model training on the SynDocDS dataset achieves an enhancement in the PSNR and SSIM, increasing them from 23.00 dB to 25.70 dB and 0.959 to 0.971 on average. In addition, the experiments demonstrated that our DSFN clearly outperformed other networks across multiple metrics, including the PSNR, the SSIM, and its impact on OCR performance.

RGB color constancy using multispectral pixel information.

Multispectral imaging is a technique that captures data across several bands of the light spectrum, and it can be useful in many computer vision fields, including color constancy. We propose a method that exploits multispectral imaging for illuminant estimation, and then applies illuminant correction in the raw RGB domain to achieve computational color constancy. Our proposed method is composed of two steps: first, a selected number of existing camera-independent algorithms for illuminant estimation, originally designed for RGB data, are applied in generalized form to work with multispectral data. We demonstrate that the sole multispectral extension of such algorithms is not sufficient to achieve color constancy, and thus we introduce a second step, in which we re-elaborate the multispectral estimations before conversion into raw RGB with the use of the camera response function. Our results on the NUS dataset show that an improvement of 60% in the color constancy performance, measured in terms of reproduction angular error, can be obtained according to our method when compared to the traditional raw RGB pipeline.

Generative models for color normalization in digital pathology and dermatology: Advancing the learning paradigm

Color medical images introduce an additional confounding factor compared to conventional grayscale medical images: color variability. This variability can lead to inconsistent evaluation by clinicians and the misinterpretation or suboptimal learning process of automatic quantitative algorithms. To mitigate the potential negative consequences of color variability, several color normalization strategies have been developed, proving to be effective in standardizing image appearance. In this paper, we present a novel paradigm for color normalization using generative adversarial networks (GANs). Our method focuses on standardizing images in the field of digital pathology (stain normalization) and dermatology (color constancy), where high color variability is consistently observed. Specifically, we formulate the color normalization task as an image-to-image translation problem, ensuring a pixel-to-pixel correspondence between the original and normalized images. Our approach outperforms existing state-of-the-art methods in both the digital pathology and dermatology fields. Extensive validation using public datasets demonstrate the effectiveness of our color normalization results on entirely external test sets. Our framework exhibits strong generalization capability on unseen data, making it suitable for inclusion in the pipeline of automatic quantitative algorithms to reduce color variability and improve segmentation and/or classification performance. Lastly, we provide the source code of our models to encourage open science.

Multi-scale color constancy based on salient varying local spatial statistics

AbstractThe human visual system unconsciously determines the color of the objects by “discounting” the effects of the illumination, whereas machine vision systems have difficulty performing this task. Color constancy algorithms assist computer vision pipelines by removing the effects of the illuminant, which in the end enables these pipelines to perform better on high-level vision tasks based on the color features of the scene. Due to its benefits, numerous color constancy algorithms have been developed, and existing techniques have been improved. Combining different strategies and investigating new methods might help us design simple yet effective algorithms. Thereupon, we present a color constancy algorithm based on the outcomes of our previous works. Our algorithm is built upon the biological findings that the human visual system might be discounting the illuminant based on the highest luminance patches and space-average color. We find the illuminant estimate based on the idea that if the world is gray on average, the deviation of the brightest pixels from the achromatic value should be caused by the illuminant. Our approach utilizes multi-scale operations by only considering the salient pixels. It relies on varying surface orientations by adopting a block-based approach. We show that our strategy outperforms learning-free algorithms and provides competitive results compared to the learning-based methods. Moreover, we demonstrate that using parts of our strategy can significantly improve the performance of several learning-free methods. We also briefly present an approach to transform our global color constancy method into a multi-illuminant color constancy approach.

Invited Session III: Diversity in chromatic processing across the animal kingdom: Color vision in stomatopod crustaceans.

Stomatopod crustaceans, commonly known as mantis shrimp, have perhaps the most unusual color-vision systems of any animals. The uniqueness is possible because stomatopods have compound eyes. Here, each unit, or ommatidium, acts as an independent visual detector, with its own corneal lens, internal optics, and set of photoreceptors. Ommatidia tuned to different wavelengths can be individually placed in the eye to build unusual color systems. In mantis shrimps, the receptors responsible for color vision are limited to six parallel rows of ommatidia that together form an equatorial belt, called the midband. Various receptors in these ommatidia are tuned to eight narrow-band spectral channels in the visible spectrum plus up to four additional ultraviolet channels. Thus, there is a total of twelve different color receptors for color vision. How these color channels are analyzed in the complex set of optic lobes existing behind the retina is only partly understood. It appears that stomatopods use both opponent and labelled-line color channels. Oddly, these animals appear to have limited ability to discriminate between spectral lights, but they have outstanding color constancy. Color vision, and color processing, in stomatopods is probably unlike that of any other animal group.

Misidentifying illuminant changes in natural scenes due to failures in relational colour constancy.

The colours of surfaces in a scene may not appear constant with a change in the colour of the illumination. Yet even when colour constancy fails, human observers can usually discriminate changes in lighting from changes in surface reflecting properties. This operational ability has been attributed to the constancy of perceived colour relations between surfaces under illuminant changes, in turn based on approximately invariant spatial ratios of cone photoreceptor excitations. Natural deviations in these ratios may, however, lead to illuminant changes being misidentified. The aim of this work was to test whether such misidentifications occur with natural scenes and whether they are due to failures in relational colour constancy. Pairs of scene images from hyperspectral data were presented side-by-side on a computer-controlled display. On one side, the scene underwent illuminant changes and on the other side, it underwent the same changes but with images corrected for any residual deviations in spatial ratios. Observers systematically misidentified the corrected images as being due to illuminant changes. The frequency of errors increased with the size of the deviations, which were closely correlated with the estimated failures in relational colour constancy.

LSSVM-CC: ensemble algorithm of color constancy using least square support vector machine

LSSVM-CC: ensemble algorithm of color constancy using least square support vector machine

Impact of artificial intelligence-based color constancy on dermoscopical assessment of skin lesions: A comparative study.

The quality of dermoscopic images is affected by lighting conditions, operator experience, and device calibration. Color constancy algorithms reduce this variability by making images appear as if they were acquired under the same conditions, allowing artificial intelligence (AI)-based methods to achieve better results. The impact of color constancy algorithms has not yet been evaluated from a clinical dermatologist's workflow point of view. Here we propose an in-depth investigation of the impact of an AI-based color constancy algorithm, called DermoCC-GAN, on the skin lesion diagnostic routine. Three dermatologists, with different experience levels, carried out two assignments. The clinical experts evaluated key parameters such as perceived image quality, lesion diagnosis, and diagnosis confidence. When the DermoCC-GAN color constancy algorithm was applied, the dermoscopic images were perceived to be of better quality overall. An increase in classification performance was observed, reaching a maximum accuracy of 74.67% for a six-class classification task. Finally, the use of normalized images results in an increase in the level of self-confidence in the qualitative diagnostic routine. From the conducted analysis, it is evident that the impact of AI-based color constancy algorithms, such as DermoCC-GAN, is positive and brings qualitative benefits to the clinical practitioner.

ARiRTN: A Novel Learning-Based Estimation Model for Regressing Illumination.

In computational color constancy, regressing illumination is one of the most common approaches to manifesting the original color appearance of an object in a real-life scene. However, this approach struggles with the challenge of accuracy arising from label vagueness, which is caused by unknown light sources, different reflection characteristics of scene objects, and extrinsic factors such as various types of imaging sensors. This article introduces a novel learning-based estimation model, an aggregate residual-in-residual transformation network (ARiRTN) architecture, by combining the inception model with the residual network and embedding residual networks into a residual network. The proposed model has two parts: the feature-map group and the ARiRTN operator. In the ARiRTN operator, all splits perform transformations simultaneously, and the resulting outputs are concatenated into their respective cardinal groups. Moreover, the proposed architecture is designed to develop multiple homogeneous branches for high cardinality, and an increased size of a set of transformations, which extends the network in width and in length. As a result of experimenting with the four most popular datasets in the field, the proposed architecture makes a compelling case that complexity increases accuracy. In other words, the combination of the two complicated networks, residual and inception networks, helps reduce overfitting, gradient distortion, and vanishing problems, and thereby contributes to improving accuracy. Our experimental results demonstrate this model's outperformance over its most advanced counterparts in terms of accuracy, as well as the robustness of illuminant invariance and camera invariance.

A New Interpretation of the Physical Color Theory Based on the Descartes´ Rotation Energy of Visible, Ultraviolet and Infrared Photons

During the past four hundred years the Newtonian and Goethean schools collected many experimental observations on the formation of colors. The situation became more complicated after the experiments of Land who documented that the classical color theory (= colors correspond to exact wavelength of light) is valid completely in the dark surroundings only. We are at the crossroads to find the boundary between the physical properties of colors and the perception of those photons in the retina and the brain. Therefore, the more general physical color theory should interpret those color effects where the classical physical color theory fails. As a potential candidate we present the overlooked Descartes´ color theory based on the rotation energy of visible, ultraviolet, and infrared photons. The rotation energy of colors was defined as E = hν570*(λx/λ570) where index describes the wavelength of photons in nanometers. This new mathematical description of colors enables to newly interpret situations where the average rotation energy of all reflected photons determines the color impression. The colors formed behind the triangular prism could be interpreted as the lateral diffusion of the rotation energy of ultraviolet and infrared photons (= called by Old Masters as the interplay of darkness with the light) through the field of refracted visible photons and with the modification of their rotation energy. The white color can be interpreted as the constructive interference of red, green, and blue colors with the constructive angle cos (120°) = -0,5. The black color can be interpreted as the destructive interference of cyan, magenta, and yellow colors with the destructive angle cos (180°) = -1. For the case of illumination with a range of wavelengths the resulting color is determined from the average rotation energy of all reflected photons – a model for the interpretation of the color constancy.

Underwater image enhancement utilizing adaptive color correction and model conversion for dehazing

Underwater imaging is typically affected by absorption and scattering, which can lead to color distortion and image blurring. However, no universal color constancy algorithm for correcting underwater images exists due to the wide variation in underwater scenes. Furthermore, methods based on learning to improve underwater image quality often suffer from a lack of ground truth, which can compromise the validity and authenticity of restored images. In this study, we propose an adaptive underwater image enhancement technique that corrects image color and reduces haze using a valid dataset with ground truth. Specifically, we first use hue channel statistics from underwater images to build a dataset of color-corrected images from different underwater scenes. We use this dataset to train an Unet-like network for adaptive color correction. After color correction, underwater images often exhibit characteristics similar to hazy terrestrial images. We refer to this phenomenon as “model conversion”. The hazy terrestrial images have corresponding ground truth. Therefore, we train a Transformer-like network with hazy terrestrial image datasets to remove haze in underwater images. Our method is more robust for different underwater scenes and elegantly solves the problem of lacking ground truth. A series of experiments demonstrate that our method achieves superior performance compared to state-of-the-art methods in terms of both visual quality and quantitative metrics.

A color constancy based flower classification method in the blockchain data lake

A color constancy based flower classification method in the blockchain data lake